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ABSTRACT 

The Multilevel Evaluation Systems Project is 
exploring the requirements for information systems that could help 
teachers and administrators sort through, analyze, and apply 
comprehensive information about their students, community, 
instructional processes, and outcomes to improve their schools. 
Toward this end, a multi-disciplinary literature review was conducted 
to identify guiding principles for system design and implementation. 
The direction of the literature review was shaped by Lucas' 
observation that the major reason information systems fail is that 
designers concentrate on technical aspects while they overlook users 1 
organizational behavior and needs. The major research literature 
covered includes teacher planning, evaluation utilization, 
information representation and decision making, computers and user 
friendliness, management information systems, and cognitive 
psychology. The rationale for seleccing these major areas is 
described, and the conclusion summarizes the major principles and 
implications suggested by the research. The summary includes 10 
guiding principles and an accompanying checklist of planning and 
review questions for the design and implementation of a maximally 
useful information system. (63 references) (GL) 
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Increasing the utility of Information Systems in 
Schools: Lessons from the Literature 
Introduction 

Schools and the teachers anu administrators within 
them are bombarded with vast amounts of information 
that is potentially useful in instructional planning 
and school improvement. Students' test scores , 
attendance, grades, behavior, results of other 
measures, information about curriculum goals and 
standards, special programs, student and community 
interests, and parent surveys are just a sample of the 
data that is regularly collected and/or transmitted to 
schools. Teachers, at least implicitly, are expected 
to incorporate their knowledge of this wide array of 
information into coherent and effective instructional 
plans. Expectations for school level planning 
similarly abound as schools attempt to meet increasing 
legislative demands for accountability, responding to 
repeated waves of district, state, and national 
findings in combination with state and local mandates 
and local school priorities to direct their school 
improvement agendas. 

Despite implicit expectations, however, the 
availability of information in schools has outstretched 
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its utility. At best, such information represents an 
important resource base for decision-making and action; 
at worst it represents a disorganized and overwhelming 
set of unknown messages. , Can the potential power of 
information be marshalled? other sectors have been 
quick to apply the computer's capacity to store, 
organize, and analyze information to create 
multi faceted management information systems to support 
their decision-making. The availability of low-cost, 
high-power computer technology coupled with user 
friendly software systems, make such information 
systems a reasonable possibility in schools as well. 

The Multilevel Evaluation Systems project is 
exploring the requirements for information systems that 
could help teachers and administrators sort through, 
analyze, and apply comprehensive information about 
their students, community, instructional processes, and 
outcomes to improve their schools. Toward this end, a 
multi-disciplinary literature review was conducted to 
glean guiding principles for system design and 
implementation. The direction of the literature review 
conducted was shaped by Lucas' (1975) observation that 
the major reason information systems fail is that 
designers concentrate on technical aspects while they 
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overlook users' organizational behavior and needs. 
Hence, rather than iocusing on the technical details of 
developing an educational information system, this 
paper concentrates on the organizational behavior and 
needs of teachers and principals and how evaluative 
information can be best developed and presented to 
match those behaviors and needs. 

The major research literatures covered in this 
paper include Teacher Planning, Evaluation Utilization, 
Information Representation and Decision-Making, and 
Computers and User Friendliness. The research 
literatures on Management Information Systems (MIS) and 
Cognitive Psychology were also investigated. Since the 
findings from these investigations reiterated and lent 
further support for the findings reported in the other 
four categories, the decision was made to incorporate 
the MIS and Cognitive Psychology research into these 
other four sections. 

Our rationale for choosing these four major topics 
is described here to help the reader see the 
relationship between the topics and improving an 
instructional information system's sensitivity to 
users' organizational behaviors and needs. 

Teacher planning . The purpose for the section on 



9 

ERJC 



5 



teacher planning is to better understand teachers' 
routine planning and decision-making processes so that 
an information system can be optimally designed to 
naturally fit into and accommodate those routines. 

i 
I 

Gringas and McLean (1981) findings reinforce the 
importance of this issue. They found that many 
designers who claim to have a user orientation still 
develop programs that do not meet user expectations. 
Not surprisingly, they alse found that the designers' 
images of actual users were significantly different 
from the users' self-images. Useful information 
systems clearly need to be grounded in users' actual 
needs rather than in designers' assumptions about those 
needs. To avoid faulty images, this section of the 
paper suggests some of the issues that will be of 
special concern to teachers. However, to insure that 
users' needs are truly being incorporated :nto the 
design of an information system, potential users will 
have to be continually consulted and their 
recommendations heeded during all phases of the 
system's design and implementation. 

Evaluation utilization. The section of the paper 
on evaluation utilization concentrates on the factors 
which have been found to influence the utility and 
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influence of evaluation data in educational decision- 
making. Following Lucas' (1975) observation that many 
intended users do not actually use information provided 
because of inadequacies in the system's design or 
implementation, the section on evaluation utilization 
addresses what aspects of design and implementation are 
necessary for encouraging information use. 

Information represent ation and decision-making. 
How information can be represented in a manner that is 
both easily comprehended and directly applied is the 
focus of the third section of the paper. Lucas (1975) 
points out that many information systems fail because 
users do not understand the systems' output or are 
overwhelmed with the amount of information with which 
they are presented. Research on information 
representation can help avoid such failures by 
suggesting principles of effective displays. 

Computers and user friendliness . The section of 
the paper on computers and user friendliness offers 
several concrete suggestions for creating systems that 
are both easily learned and easily operated. The 
central admonition of this section is similar to that 
of the section on evaluation utilization; that is, to 
increase utility, be sensitive to the user. However, 
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because of the number and specificity of detailed 
suggestions made in the literature, a separate section 
on user friendliness seemed warranted. 

The conclusion of the paper summarizes the major 
principles and implications suggested by the research. 
The summary is conceived as a checklist of planning and 
review questions that can be used to insure the design 
and implementation of a maximally useful information 
systems . 

Teacher Planning 

Introduction 

Planning is an important aspect of a teacher's 
work. Teachers report that they spend between 10-20 
hours per week in planning, and much of that is during 
non-school hours (Clark & Yinger, 1979) . Planning 
decisions influence the content, materials, social 
climate, grouping, and activities experienced in the 
classroom (Shavelson, 1982) . According to Yinger 
(1980) there are five basic levels of teacher planning. 
They are yearly planning (selection of general 
materials, placement of pupils, and sequencing and 
organizing of instruction tor the entire school year) , 
term planning (determining weekly schedules and unit 
activities within the time, framework of the term) , unit 
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planning (laying out activities that are to be part of 
instructional units in specific areas of subject 
matter) , weekly planning (focusing on specific 
activities and the schedule for those activities) , and 
daily planning (making last minute changes or 
preparations for the day's activities) . The proposed 
MLES information system is intended to aid teachers and 
principals with yearly, term, and unit planning that is 
focused on larger chunks of curricular content or 
student behavior. To better understand how an 
information system can help teachers and principals 
with these types of planning, this section of the paper 
summarizes recent theory and research concerned with 
teacher planning and suggests their implications for 
the design of effective information systems. 
Rasearch on Teacher Planning 

Research and theory in cognitive science indicate 
the importance of understanding how teachers think, 
their existing schema, and their modes of information 
processing. However, research on teachers' thought 
processes has focused almost exclusively on particular 
learning episodes and has neglected broader types of 
planning activities* (Clark & Peterson, 1986) . In other 
words, research about the types of planning the MLES 
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intends to affect is practically non-existent. 
Nonetheless, findings from related areas of teacher 
planning research can offer insights on teachers' 
orientations and general planning dispositions. The 
related areas surveyed by this review are how and what 
teachers consider when planning, the impact of time of 
year on planning, the influence of teacher expectations 
on planning, and the possible negative effects of 
planning. 

How and what teachers consider when planning. 
Yinger's cyclical teacher planning model (Yinger, 1978) 
offers one potentially useful model for understanding 
how teachers plan. This model uses a cognitive 
approach to describe how teachers interpret and 
elaborate on information during the planning process 
and how teachers' past experience and knowledge will 
affect that process. According to this model, there 
are three stages during teacher planning: Problem- 
finding; Problem Formulation/Solution; and 
Implementation, Evaluation, and Rout inizat ion. 

In the Problem-finding stage, the general planning 
task is translated into a specific planning problem 
called the problem-conception. This is accomplished by 
teachers considering in a cyclical fashion their own 
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knowledge and experience, their teaching goal 
conceptions (their expectations and theories about 
effective teaching) , available materials, and other 
external factors such as the teaching environment and 
school organization (physical characteristics of 
classroom and the school, the number of students in the 
class, the length of the school day, and the teacher's 
relationship with the principal and other teachers) , 
curriculum and resources (guidelines inherent or 
explicit in school or district objectives and programs, 
personnel, and materials supplied to the school for 
teaching certain subjects) , and pupil characteristics 
(students' background and teacher's judgment of 
students' ability , maturity, attention span, and 
ability to work in a group) . During this stage of 
teacher planning, an information system can be 
potentially useful by increasing teachers' base of 
knowledge. This might Le done by generating reports 
about school and district requirements, available 
instructional and other resources, and pupils' and 
theii parents' characteristics, and by suggesting 
measures teachers can use to gather more information 
about any of these areas. 

An information system may be particularly useful 
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during the Problem-finding stage if it is able to 
generate reports that are sensitive to an individual's 
or a school's teaching goals, such goals incorporate 
teachers' expectations for cognitive and affective 
outcomes, as well as the teachers' implicit and 
explicit theories about the nature of effective 
teaching. Munby (1983) also found that teachers' 
implicit theories (principles) and beliefs about 
teaching are tied to their teaching goals and 
significantly influence the effectiveness of any 
curricular, organizational, or instructional change. 
He found that most teachers' principles and beliefs 
could be categorized according to the following 
constructs: student learning and developmental goals 
(curriculum goals? extra-curricular, academic, and 
personal goals; and subject matter principles) , teacher 
autonomy and authority (efficacy, degree of control 
over curriculum, evaluation, etc.), teachers' needs 
(e.g., teachers' needs for order, for interpersonal and 
collegial relations, and for increasing professional 
knowledge), students' needs and limitations (students' 
personal and academic needs, student differences, and 
student limitations) , and conceptions of factors which 
facilitate learning (motivation, self-esteem, 
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questioning, instructional variety, application and 
transfer, classroom climate, and order) . One possible 
way of making an information system sensitive to 
teachers' teaching goals might be to have teachers 
input their more important teaching goals into the 
system and have the system recommend measures or bring 
up information that is responsive to teachers' 
priorities. 

Relevant to the kinds* of information teachers 
might find useful during the Problem-finding stage, 
Shavelson and Stern (1981) found that teachers usually 
seek the following information about their students: 
ability or achievement, class participation, self- 
concept, social competence, independence, classroom 
behaviors, and work habits. These factors, used in 
making pedagogical decisions about students and related 
to the aforementioned teaching goals, signal elements 
that might be included among an information system's 
pupil characteristics measures. 

Research by Floden, Porter, Schmidt, Freeman, and 
Schwille (cited in Potter, 1983) suggests the 
importance of specific information on district test 
results and district instructional objectives. They 
found that these factors had significant impact on 
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teachers' planning and their willingness to change the 
content of their teaching. 

The next stage of the planning process, Problem 
Formulation and Solution, .also proceeds in a cyclical 
fashion, whereby the teacher alternates between 
representing the problem, formulating partial 
solutions, and mentally trying out solutions until a 
workable plan results. This cycle can better be 
conceived as a coil in which an initial problem 
conception begins at the top and moves downward in a 

i 

circular motion through repeating cycles of developing 
and fine tuning the problem conception and a tentative 
solution, checking the solution's workability, and 
matching the tentative solution to the expectations of 
the problem conception. When the tentative solution 
does match the goals of the problem conception, this 
stage is complete, and the tentative solution is ready 
for the stage of Implementation, Evaluation, and 
Routinization. Although an information system may not 
be able to offer any direct assistance to teachers 
during the stage of Problem Formulation and Solution, 
the finding that teachers focus on the elaboration and 
refinement of a single alternative at a time does have 
potential implications for the sequence of information 
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reports and suggested planning procedures accompanying 
a system. The sequence of reports might start with 
general level reports to help users locate specific 
problems or educational variables that are of interest 
to them. Then more specific and detailed inquiries and 
reports can be generated to aid the users in clarifying 
the nature of the specific problem identified. 
Further, to be consistent with teachers' current 
routines, and thereby potentially facilitate use, 
accompanying procedures might focus on the development 
and refinement of one alternative solution at a time. 

The final stage, Implementation, Evaluation, and 
Routinization, is when the solution is tried out and 
evaluated in practice. The results of this stage add 
to the repertoire and experience of the teacher, which 
in turn become an important part of subsequent 
planning. At this stage, the information system can be 
useful by recommending, analyzing, and reporting the 
results of measures for evaluating the effectiveness of 
the activities, programs, and environmental or 
curricular changes the teachers have implemented. 

Yinger's model of teacher planning was supported 
by research (Clark & Yinger, 1979) that found teachers 
differentiated in their own minds between the Problem 
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Finding and Problem Formulation/Solution stages. This 
research also supported the notion of the planning 
process being a progressive elaboration of one idea 
rather than the development of several alternative 
ideas and then the selection of an optimal alternative 
from among these. These findings further support the 
notion that an information system and the process 
surrounding it should be structured to enable teachers 
to analyze in depth one issue at a time. 

It is interesting to note the parallels between 
Yinger's model and models supported in the management 
information literature. The latter, as presented by 
Brookes (1986) , similarly recommends that information 
systems should be targeted to specific phases of the 
decision-making process (e.g., problem recognition, 
diagnosis, generation of alternative solutions, 
evaluation of outcomes of alternatives, and judgments 
between alternatives) . Brookes also recommends that 
the information needed for recognizing situations that 
require a response should be separated from information 
that is needed to solve problems. He found that 
removal of all information of a problem-solving 
character from reports can reduce their size by 50% and 
lead to faster problem recognition (Brookes, 1986). 
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Finally, Brookes notes that the problem-solving 
information needs to still be available to the user 
upon their request. 

Impact of time of school year on planning. Clark 
and Elmore (1979) found that teachers during the first 
five weeks of schpol are primarily concerned with 
setting up the physical environment of the classroom, 
assessing student abilities, and establishing the 
social system of the classroom. By the end of the 
fourth week, the teacher has established a system of 
schedules, routines, and groupings for instruction, and 
these structural features ser<;e as a framework 
throughout the school year within which teachers plan 
particular activities and units. Joyce (1978-79) also 
found that teacher planning decisions made early in the 
academic year had a profound influence on teachers' 
planning for the remainder of the year. Finally, Clark 
and Peterson (1986) report that several researchers 
have concluded that the planning that takes place in 
the first few weeks of the school year has a powerful 
impact on the behaviors and thoughts of the student and 
the teacher throughout that school year. This body of 
research suggests that to maximize the impact of an 
information system, current information needs to be 
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available either before or at least within the first 
few weeks of the school year. Such currency might be 
accomplished by a constant updating of data or an 
updating of data every spring. 

Influence of teacher expectations on planning. It 
has been found that teachers' expectations of their 
students' abilities influence their interactions with 
those students (Brophy & Good, 1970) . Furthermore, it 
has been found that a teacher's initial expectation may 
serve as an anchor for his or her subsequent estimates 
of the student's ability (Shavelson, 1982). Dusek 
(1975) found that even when teachers know their 
expectations for students are based on false 
information, the teachers are still influenced by these 
expectations when interacting with students. This 
finding was supported by Shavelson and his associates 
who reported that initial estimators of student 
characteristics are difficult to overcome, even in the 
face conflicting evidence (Shavelson, Caldwell, & 
Izu, 1977) . 

The power teachers' expectations exert over their 
planning and behavior must be cautiously handled by the 
information system. It is important that a system not 
lead teachers to false expectations (too high or too 
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low) for their classes, making validity and reliability 
of the information provided essential. A variety of 
indicators on each variable reported by the system can 
help to strengthen the validity base. Having the 
system remind users of the fallibility of results might 
be another safeguard. Further, to aid teachers in 
overcoming false expectations they may already hold, 
conflicting evidence should be presented in a highly 
salient manner. 

Possible negative effects o f planning . Based on 
ethnographic research, McCutheon (1980) found that some 
teachers view long-range planning to be a waste of time 
because of unpredictable changes in schedules and 
interruptions. Zahorik (1970) found that planning may 
become counter-productive if teachers become single- 
minded and do not adapt their lessons to students' 
needs. Peterson, Marx, and Clark (1978) found, in a 
lab setting, that students of tutors who planned 
extensively achieved lower and had poorer attitude 
scores than students of tutors who did not plan so 
extensively. It appears that planning can be counter- 
productive if it causes teachers to be inflexible and 
unresponsive to student input and their teaching 
environments. In order to avoid potential detrimental 
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effects of planning, the planning context surrounding 
the system use should emphasize general strategies and 
target goals and objectives rather than specific, 
detailed plans of action. Also, during training and 
orientation, teachers should be made aware of the 
negative effects of strict adherence to plan? and the 
importance of modifying plans to be sensitive and 
responsive to student needs. 

General Implications from Teacher Planning Research 

The following is a listing of the general 
implications derived from the research discussed in 
this section. 

1. The system could generate reports about 
important elements in the teacher's problem conception. 
For most teachers these elements are organized in the 
following areas: school and district environment and 
organization, school and district resources, district 
requirements and tests, curriculum requirements, and 
pupil . characteristics . 

2. The system should be sensitive to and 
accommodate users' unique teaching goals and 
orientations which are based in the elements contained 
in implication number 1. The teaching goals should 
include both the teacher's expectations for students' 
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cognitive and affective outcomes, and take into account 
the teacher's theories and beliefs about effective 
teaching. 

3. Rather than helping teachers generate a list of 
alternative planning solutions, the system should help 
teachers clarify their conceptions of their planning 
problem and to investigate one tentative solution at a 
time, by using their own knowledge, experience, and 
teaching goals in combination with information about 
available materials, environmental and organizational 
factors, requirements, and pupil characteristics. 

4. The system should help teachers evaluate the 
effectiveness of activities, programs, and curricular 
or environmental changes. 

5. Pupil characteristics that might be included in 
the system's data base are achievement, class 
participation, self -concept, social competence, 
independence, classroom behaviors, work habits, and 
background characteristics. 

6. The system should be available for use by 
teachers before the school year begins or within its 
first four weeks to maximize the system's effects on 
teacher planning. 

7. The current data should be entered into the 
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system no later than a month before the beginning of 
the school year in order to match most teachers' 
critical period for planning (a few weeks before school 
starts through the first four weeks of instruction) . 

8. The system should be sensitive to initial 
expectations which may lead users to misinterpret or 
ignore important data. To promote sound expectations, 
the data should be reliable and valid, important 
findings should be supported by multiple indicators, 
and the limitations of the findings should accompany 
their reporting. The power of prior expectations might 
also be handled by having the users state their 
expectations for the data prior to finding out the 
results. The system could then clearly indicate 
whether or not those expectations are supported by the 
available data. 

9. The system should demonstrate with examples its 
value and flexibility in teacher long-range planning. 
Orientation and training should recognize that some 
teachers feel that long-range planning is a waste of 
time due to unpredictable changes in schedules and 
interruptions. To maximize utility and minimize 
problems of inflexibility, planning activities could 
focus on understanding of the problem and general 
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strategies for their solution that can be adapted to a 
variety of specific situations. 

10. Orientation and training should make teachers 
aware of the potential negative effects of strict 
adherence to plans. Over-adherence, in fact, may 
detract from their sensitivity and responsiveness to 
students' needs. 

Evaluation utilization 

Introduction 

Research on evaluation utilization usually focuses 
on one of the following definitions: (1) use of 
results to support discrete decisions and (2) use in 
the general education of decision-makers (Cousins & 
Leithwood, 1986). The MLES project is primarily 
concerned with the first definition — direct uses of 
information to help teachers and principals with their 
decision-iuaki j. To better understand how to increase 
the probability of such utilization, this section of 
the paper draws upon the recommendations of evaluation 
experts and on implications derived from the research 
literature. But before discussing how to increase the 
system's utilization, suggestions for possible uses of 
the system are described. 
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Possible Uses for mt,fs 

Among the many topics discussed in their book, 
Toward Reform of Program Evaluation , cronbach and his 
associates (1980) discuss the possible uses of 
instructional information systems. The book offers the 
possibilities of a system designed to both assess 
individual students' needs and diagnose resources that 
match those needs, make comparisons and other inquiries 
on a cl^ss and school level, and examine school 
patterns, the demographics of the communities in which 
they are based, and how funding affects students 
through different programs. Examples of uses suggested 
by this book that relate to the MLES information system 
are checking out the homogeneity or heterogeneity of 
class assignments, comparing the progress of this 
year's students with last year's students, examining 
the school's profile of achievement in reading as 
compared with math or within the reading area, and 
finding out how students are doing on decoding skills 
as compared with comprehension skills. 

Besides possibilities for an educational 
information system, there are also limits. As argued 
by Lindbolm and Cohen (1979), most decisions and 
actions are based upon "ordinary knowledge" — that 
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knowledge derived from practical experience which is 
widely shared , sensitive to context, and comprehensive. 
By contrast, . knowledge that comes from social science 
evaluations is context independent and selective rather 
comprehensive. Therefore, the knowledge obtained 
through evaluations can only serve to supplement 
ordinary knowledge and cannot be expected to replace or 
dominate such knowledge. 

Suggestions and Implications for Evaluation Utilization 

The suggestions of experts and implications from 
research in the field of evaluation utilization 
commonly relate to the following general 
characteristics of evaluations and their context: 
evaluation quality and credibility, relevance to 
information needs, communication quality and ease of 
use. timeliness, commitment and receptiveness towards 
the evaluation, political climate, and personal 
characteristics. These factors have all been found to 
be correlated and to influence in combination, how and 
to what extent evaluations are used (Cousins & 
Levinson, 1986) . 

Evaluation quality and credibility. An 
evaluation's quality is define* 3 by the sophistication 
of its methods, the intensity of evaluation activities, 
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and the type of technical approach to the evaluation, 
while its credibility is defined by the user's 
perceptions of its objectivity, believability, and 
appropriateness of the evaluation criteria (Cousins & 
Leithwood, 1986). Weiss and Bucuvalas v 1980) do not 
distinguish between evaluation quality and credibility, 
but rather consider credibility to be a part of 
evaluation quality which is defined as the technical 
quedity, statistical sophistication, objectivity of the 
researcher, and internal consistency reflected by the 
evaluation (Weiss & Bucuvalas, 1980) . Although 
evaluation credibility is equivalent to the user's 
perception of evaluation quality, credibility and 
quality can differ. For example, evaluations that 
methodologists might consider of high technical quality 
may be considered trivial or unbelievable by users and 
an evaluation perceived as credible to its users might 
not be of high technical quality to technical experts. 
Several researchers have reported that improved 
methodological sophistication was positively related to 
both direct and indirect use by decision-makers 
(Cousins & Leithwood, 1986) . They also reported that 
when evaluations were perceived by decision-makers as 
having high face validity use and the potential for use 
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appeared to be greater (Cousins & Leithwood, 1986) . 
Finally, Weiss and Bucuvalas (1980) found that 
perceived research quality was the single most 
significant factor in determining whether or not a 
decision-maker chose to use an evaluation. 

Concerned with evaluation quality within the 
context of an information system, Cooley and Bickel 
(1986) recommend that an educational information 
system's data should be current and accurate. To do 
this they advise that the system be merged with school 
and district's central data files that are used to keep 
track of daily functions such as attendance, student 
enrollment, and standardized testing. To increase the 
incentive for entering current data, they propose 
offering same day scoring for achievement tests. 
Finally, they suggest accuracy can be improved by 
building in checks for out-of-range data within the 
program and by avoiding corruption of data by shifting 
the focus of the evaluation from accountability of 
teachers and schools to local school improvement. 

One factor that affects the credibility of an 
evaluation is whether evaluation findings are 
consistent or inconsistent with the evaluation 
audience's expectations. The majority of studies 
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concerned with user expectations found that when 
evaluation findings were consistent with decision-maker 
expectations, acceptance and utilization increased; 
incongruent findings tended to be ignored and other 
information used instead (Cousins & Leithwood, 1986) . 
Based on such findings, Alkin, Dalliak, and White 
(1979) recommend that evaluations be user oriented — 
that the* reports and areas of inquiry be sensitive to 
user expectations. 

Interestingly, Weiss and Bucuvalas (1980) found 
that results which challenge the status quo and those 
that are critical of agency practice are -Associated 
with increased direct use and increased likelihood of 
being taken into account. Although this appears to 
conflict with the findings concerned with user 
expectations, it does not. Because user expectations 
and user beliefs in the status quo's effectiveness are 
not correlated, these authors suggest that many users 
already question the effectiveness of status quo 
practices. Furthermore, Weiss and Bucuvalas found that 
when it is not politically feasible to directly act 
upon data that challenges the status quo, this type of 
data still serves the important function of directing 
the user's attention to possible changes that could be 
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implemented if the opportunity arises. 

Another factor that can detract from evaluation 
credibility is competing information — that information 
obtained from sources beyond the evaluation (e.g., 
personal observations) that conflicts with the 
evaluation data and has bearing on the decision being 
made (Cousins & Leithwood, 1986) . Research has found 
that knowledge from personal experiences , beliefs , 
values , interests , and goals are powerful competitors 
with evaluation data (Cousins & Leithwood, 1986) . 

Relevance to information needs . Relevance of the 
evaluation is defined as the extent to which the 
evaluation is geared to the audience's information 
needs (Cousins & Leithwood, 1986) or the extent to 
which the findings are germane to the issues with which 
the user is concerned (Weiss & Bucuvalas, 1980). Most 
studies concerned with relevance and utilization have 
found that evaluations which reflected knowledge of the 
context in which the evaluations were to be used, 
sought consensus about the evaluation problem, or 
demonstrated insight into the necessary decision-making 
were associated with higher levels of use; evaluations 
that ignored these issues were associated with 
relatively low levels of use (Cousins & Leithwood, 
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1986) . 

Cooley and Bickel (1986) recommend that to 
increase the relevance of evaluations carried out by an 
information system, a system should anticipate and 
respond to information needs of potential users. In 
the case of teachers and principals, they felt these 
information needs included the ability to examine a 
wide variety of student data in a correlational manner; 
the capacity to spot new developments or patterns by 
the use of longitudinal data; the choice of examining 
data at student, classroom, and school levels (this 
system is being designed to handle the last two levels 
as well as district wide comparisons) ; the ability to 
conduct inquiries designed by the user; and the 
provision of information to teachers about 
instructional materials that would be particularly 
helpful for students • 

Patton (1987), a strong supporter of designing 
evaluations relevant to the user, recommends the 
involvement of potential users from the beginning of 
the evaluation process in order to incorporate the 
user's interests and informational needs. Further, he 
suggests using an identifiable, real, specific, and 
caring group of potential users on which to base 
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evaluation designs, rather than a general or abstract 
audience. Since the aim of the MLES project is to 
introduce the system to various school communities, 
development might use research on teacher information 
needs as a general framework for user information 
needs, but then customize that framework to the 
specific needs of the individuals who make up various 
user sites. 

Communication quality and ease of use. 
Communication quality is determined by how clearly and 
orderly the evaluation results are reported and the 
breadth of dissemination of the results. Research 
indicates that evaluation presentations or reports that 
are broadly framed, offer comprehensive 
recommendations, and use nontechnical language 
contributed to higher decision impact scores, improved 
readability, and greater awareness and appreciation of 
results (Cousins & Leithwood, 1986) . Studies also 
found that dissemination breadth (e.g., reports geared 
to the public vs. scholarly journals) resulted in 
higher utilization scores (Cousins & Leithwood, 1986). 

One finding that reflects the importance of both 
communication clarity and ease of use is that 
utilization increases when reported results require 
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little to no transformation in order to be practically 
implemented (Weiss & Bucuvalas, 1980). One way of 
accomplishing this might be to link evaluation results 
with possible forms of action. 

How easily a user can communicate which inquiries 
he or she would like to conduct is another important 
aspect of ease of use. Cooley and Bickel (1986) make 
the following recommendations for improving the ease of 
use of an information system: the system should be 
menu-driven, it should be able to produce screen 
displays as well as printouts, and it should use an 
easily learned set of commands. They also suggest that 
the system should have the capability to transfer files 
between school sites and a central district computer 
and that a scanner which provides test scoring services 
and automatically updates students' files with new 
scores should be available at every school site where 
the systam is to be used. 

Timeliness . Timeliness is a matter of decision- 
makers receiving the evaluation results when the 
information can be used to influence the decision being 
made. The majority of the research concerned with this 
factor has found that timely provision of evaluation 
results was positively related to utilization (Cousins 
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& Leithwood, 1986) . Evaluation results were also 
reported to be of most use at early stages in the 
decision-making process. 

User comm itment and receptiveness . Commitment and 
receptiveness towards evaluations are determined by the 
attitudes of the decision-makers towards evaluation, 
their organizational resistance, and their receptivity 
towards changes. It was found that user involvement in 
the evaluation process contributed to high levels of 
use of the evaluation results in decision-making. 
Lucas (1975) suggests that user involvement in the 
design and implementation of a system is the key for 
making a system succeed. He reports that user 
involvement results in favorable user attitudes and 
perceptions of information systems, which in turn lead 
to high levels of use. Lucas' recommendation to 
involve the user appears repeatedly throughout the 
Management Information System (MIS) literature as a 
solution to the problems associated with use and 
nonuse. 

Strategies for involving user; that also come from 
the MIS literature include the following (Robey & 
Markus, 1984) : 

1. "Steering Committees" involve top management in 



ERIC 



33 

« 



32 



system design by having them review proposed systems. 

2. "Information Requirements Analysis" uses a 
problem solving approach to determining what 
information is needed in making decisions, as opposed 
to merely asking users what they want. 

3. "Prototype" allows users to try out an 
experimental system and suggest changes* 

Specifically related to teachers' use of 
information systems, King (1987) reports that there are 
three attitudes teachers hold that interfere with their 
receptiveness towards such a system: pessimism towards 
making far reaching changes of any kind in classroom 
practices, skepticism towards computer innovations, and 
disbelief that instructional information systems as 
they are currently conceived will provide useful 
information for teachers. Kiuy (1987) also recommends 
ways in which an information system could be better 
received by teachers: involve all teachers in the 
development and implementation process, create a 
professional support system for teacher-users, and 
leave teachers' clinical responsibilities alone. 

Finally, several researchers found that advocacy 
by the evaluator of his or her results and forcefulness 
of communication is associated with greater use 
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(Cousins & Leithwood, 1986) . Akin to taking 
responsibility for fostering commitment among u^ers, 
Bank and Williams' (1981) findings about idea champions 
have similar implications. When evaluators emphasized 
that the evaluations were important activities and had 
important implications for the user, use increased 
(Cousins & Leithwood, 1986) . 

Political climat e. Dimensions of political 
climate include the dependence of the decision-makers 
on external sponsors, the user's and evaluator's 
political orientations, and inter- and intra- 
organizational rivalries, budget fights, and power 
struggles (Cousins & Leithwood, 1986) . Not 
surprisingly, evaluation use increases when its 
findings can be politically beneficial for the user and 
decreases when the information can be politically 
harmful . 

Cronbach jind his associates (1980) warn against 
using evaluations as a form of accountability for 
teachers* They argue that when evaluations are used to 
look back at what was done with the intent to apportion 
responsibility among teachers or program managers, the 
number of suggestions for improvements is reduced. 
Furthermore, the combination of accountability ana 



standardized tests leads to a narrowing of curricula 
and promotion of drill and practice. 

Weiss (1972) points to the possibilities of 
building a positive political climate. Sh^ recommends 
that administrators and program managers (as well as 
teachers) be involved in the evaluation process, she 
believed this would help change the image of evaluation 
from "critical spying" to collaborative effort to 
understand and improve. 

Another recommendation comes from Cooley and 
Bickel (1986) who suggest that in order to create a 
healthier political climate a system should secure 
student and teacher privacy with the use of passwords. 

Personal characteristics. Personal 
characteristics are defined in terms of the decision- 
maker's organization roles, information processing 
style, organizational experience, and social 
characteristics. While years of experience does not 
make a difference with teachers' use of evaluation for 
instructional judgments; higher levels of leadership, 
caring, and interest; more skills and initiative; and 
internal versus external locus of control are 
associated with increased use (Cousins & Leithwood, 
1986) . 
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Increasing the user's knowledge about evaluation 
is another way to affect utilization. Patton (1987) 
found that increasing decision-makers' knowledge about 
evaluation in general can contribute to greater use of 
evaluation data and a greater likelihood that users 
will continue to seek out evaluation data in the 
future, 

general Implications 

The following is a listing of general implications 
from the research and recommendations of experts, 
discussed in this section of the paper, for increasing 
the likelihood of MLES use by educators for making 
classroom/ school, and district-wide decisions. These 
implications are organized into two categories: those 
related to the technical design of information systems, 
and those related to the process surrounding their 
design and use. 

Implications for technical design. 

1. The system should offer the user a menu of 
possible inquiries based upon suggestions from the 
literature and the specific needs of target users. 

2. Data must be accurate and current. To this 
end, the system can be merged with school and district 
central data files that are used to keep track of daily 
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functions such as attendance, student enrollment, and 
standardized testing. Furthermore, the incentive for 
entering data can be increased by offering same day 
scoring for achievement tests. Finally, accuracy can 
be improved by building in checks within the program 
for out-of-range data. 

3. The system should be sensitive to and 
accommodate users' expectations, and should clearly 
report whether those expectations are or are not 
supported by the available data. 

4. Analysss should include the ability to examine 
a wide variety of student data in a correlational 
manner; the capacity to spot new developments or 
patterns by the use of longitudinal data; the choice of 
examining data at a classroom, school, or district wide 
level; the ability to conduct inquiries designed by the 
user; and the provision of information to teachers 
about the instructional materials that would be 
particularly helpful given a particular finding. 

5. To improve clarity of communication, the system 
should use simple, nontechnical language to report 
evaluation results and should require little 
transformation to be applied to educational decision- 
making. 
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6. The system must be easy to use. Methods to 
accomplish this include the following: reducing the 
amount of memory required by the learner to operate the 
system (e.g., menu driven, and easily learned commands), 
allowing for the trans* •' f files between school sites 
and a central district ater, and making available 
at every school site wt ne system is to be used a 
scanner that provides test scoring services and 
automatically updates students' files with new scores. 

7. The system must secure necessary privacy of 
information by using passwords to keep out unauthorized 
users . 

Implications for development, training, and 
support . 

1. To increase user commitment and receptiveness 
towards the system, users should be involved in 
development and implementation, encouraged to design 
their own inquiries and test their own expectations, 
and offered improvement implications that can be 
derived from the data. Further, training should 
attempt to improve users' attitudes towards evaluations 
and should organize a procedure for users to receive 
post-training assistance on how to use the system and 
its results. 
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2. Users must feel that the evaluation reports 
generated by the system are accurate, based on 
reasonable criteria and the result of objective study, 

3. Someone within the local setting needs to 
actively advocate the usefulness of the information 
system's capabilities and the importance of its results 
to users. 

4. The system must be capable of answering the 
queries that users deem relevant. The development 
process might start with user information needs based 
on research on teacher and administration information 
needs, but also must customize each system to the 
specific needs of specific target users. This 
customization might be conducted on the basis of 
information gathered by questioning a range of 
potential users on their specific information needs. 

5. Data must be timely, current and available, for 
teachers and principals at the early stages of their 
decision-making . 

6. The system should recognize and balance the 
power associated with information. The data should not 
be used for holding teachers and programs accountable. 
Such uses may lead to reductions in suggestions for 
improvements, a narrowing of curricula when 
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standardized tests are involved, and corruption of the 
data by those who are being held accountable. 
Furthermore, the information this system generates 
should not be held in the hands of only central 
officials, but should be available to all individuals 
the information affects. 

7. Training should encourage teachers and 
principals to believe that better decisions are within 
their control with the use of the information system. 

8. Training should also cover information about 
the evaluation process in general as well as specific 
uses of the system's data. 

9. To affect future utilization, training might 
encourage teachers and principals to consider findings 
which challenge the status quo and to imagine their 
implications for the future, even if they are not 
immediately feasible. 

Information Representation and Decision-Making 
Introduction 

The way numerical data are presented affects a 
user's comprehension and use of that data (MacDonald- 
Ross, 1977). Washburne (1927) found that both the 
visual and logical arrangement of data have an 
important effect upon recall and comprehension of the 
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information, in order to choose formats that 
facilitate user comprehension and use of data, it is 
necessary to understand the cognitive limitations and 
preferences of users as they relate to display formats. 
This section of the paper begins with a discussion of 
how information representation affects user information 
processing. This section also addresses the questions 
of what format to choose given a specific task, and how 
to render a particular format competently. Although 
the findings that come from empirical research on 
display formats may be helpful for display designs, 
these findings can never replace the expert designer's 
experience, intuition, critical skill, and trial of 
alternative solutions (MacDonald-Ross , 1977). 
Therefore, it may be beneficial for the development of 
the system to have an expert review the formats 
designed and to try out various alternatives with 
users . 

Cognitive Limitations and Pr eferences of the User 

To facilitate teachers' and principals' use of 
data, MLES must accommodate their cognitive limitations 
and preferences. The following are brief descriptions 
of two limitations and two preferences related to the 
comprehension and use of data, and their implications 
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for the MLES . 

Amount of information . Miller (1956) found that 
humans are only capable of retaining about seven (plus 
or minus two) chunks of information in short-term 
memory. Therefore, a good format should allow the user 
to organize and conceptualize the information in terms 
of a small number of chunks or categories. 

User misunderstanding o f statisti cal properties of 
data . Remus and Kottemann * (1986) found the following 
four errors to be the most common mistakes made when 
individuals are asked to interpret statistical 
findings: 

1. Users assume upward or downward trends are 
occurring when in fact the variations they perceive are 
random and not persistent. 

2. Users may give too much weight to 
characteristics of small samples and assume them to be 
representative of the population from which they are 
drawn. 

3. Users sometimes erroneously make inferences 
about future events based on the occurrence of past 
events that were independent of each other (Gambler's 
fallacy) . 

4. When a decision-maker is faced with several 
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sources of uncertainty simultaneously, he or she may 
simplify the problem by ignoring or discounting some of 
the uncertainty. Statistically naive users need to be 
protected against such misinterpretations. Among the 
possibilities for safeguards are having a system 
suppress analyses or reports which are inappropriate 
(e.g., those based on too small an n) , having a system 
warn users against misinterpretations in those cases 
where such errors are likely to occur, and/ or having a 
system directly interpret the data as it is reported. 

User strategies for pr ocessing information . 
Bettman and Kakkar (1977), and Ghani (1981) found that 
subjects' strategies for using information are based on 
the information that is most easily processed. Slovic 
and Lichtenstein (1968b) also found that people res'.st 
making even simple transformations of information. 
Slovic (cited in Ghani, 1981) further found that people 
tend to discount or ignore information that requires 
inference from the explicit display. Instead, people 
tend to use only the information that is explicitly 
displayed and will use it only in the form in which it 
is displayed. Therefore, a good display should present 
the information in the most straightforward and 
applicable form possible. 
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Format consistency . Users tend to resist changes 
in information representation styles (Ghani, 1981). 
Therefore, throughout the system the types of graphs 
and tables displayed should remain constant. 
Choosing a Format 

No matter what format is chosen, it is important 
to remember a point made by Weintraub (1967) — the 
skills of interpreting any types of graphs must be 
learned. Furthermore , consideration should be given to 
whether the information should be represented in a 
quantitative format or would be better communicated 
through text or verbal charts. Below are findings that 
relate to the questions of whether or not to use 
numeric data and if so, in what format. 

Numeric versus adjectival information. Numeric 
data are numbers reported as numerals or words (10 or 
ten) . Nonnumeric data are adjectival and qualitative 
descriptions (e.g., extremely large or substantial 
decline) (Bell, 1984). Scammon (1978) found that 
subjects receiving adjectival information had more 
accurate aided recall and comprehended the overall 
meaning of adjectival information better (e.cj., fair, 
good, and poor) than information presented as 
percentages. His explanation for this f indi ig is that 
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adjectival descriptions are often inherently evaluative 
in nature, making evaluation tasks easier since some of 
the information processing is already done. Scammon 
also found that subjects were more satisfied and less 
in need of more information with percentage scores as 
compared to adjectival information. This finding 
raises questions about the willingness of subjects to 
accept adjectival information even if it leads to 
better comprehension. 

Cherry (1966) also reports that users perceive 
numeric messages to be based on measurement and to be 
more precise than nonnumeric messages (adjectives) even 
when this is not the case. Again, the power of numeric 
information to portray itself as precise can be very 
misleading when the information is not of a precise 
nature. Another strength numerical data has over 
adjectival data is that it allows for easier 
comparisons across values in a display (Russo & Dosher 
as cited by Ghani, 1981) . The conflict of providing 
users with information in a form that will be more 
accurately understood (adjectival) or in a form that 
will be more readily accepted and more easily compared 
(numerical) will have to be resolved by development. 

When both numerical data and text are involved, 
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Washburne's research (1927) suggests that numerical 
data should not be presented embedded within text if 
more than two items are to be presented. Instead, 
separate tables or displays should be used. 

Formats and their functions . When considering 
which format to choose, one should be aware that no one 
graphic format proves universally superior and that 
each format has its own domain of application 
(MacDonald-Ross, 1977) . Washburne compared the effects 
of using prose, tables, pictographs, and bar charts and 
reached the following conclusion — there is no graphic 
format more effective in all respects than all other 
formats, but rather, different formats of data foster 
the learning of different types of information. 

Vernon (1950) and Washburne (1927) found that 
particular types of graphs are most effective at 
representing particular types of comparisons and 
trends. They found that (1) bar graphs are best for 
static comparisons, (2) line graphs are best for 
dynamic comparisons i trends), and (3) pictographs give 
users rapid and striking general impressions. 
Furthermore, Schutz (1961b) found that a single graph 
with multiple lines was better at representing multiple 
trends than multiple graphs with single lines. 
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Finally, Croxton and Stein (1932) found that bar charts 
facilitated easier and more accurate comparisons than 
pie charts, segmented bar charts, or three dimensional 
figures (cubes) . 

When presenting exact numbers, tables are one of 
the most important formats because of their compact 
nature. However, they have a serious weakness — they 
are abstract and therefore require more processing by 
the user (MacDonald-Ross , 1977) . Thus, the choice of 
using a table involves a trade-off between exactness, 
compactness, and ease of use. 

Pegigning Effective Formats 

The following are general and specific 
recommendations about how to design effective formats. 

Headings . Headings provide an explicit structure 
which assists the reader (user) in integrating the 
information as he or she reads. To be helpful, 
headings need to be informative and unambiguous 
(Wright, 1977) . Headings are more easily located when 
not in all capitals (Poulton, 1967) . Headings for 
graphs may be one way of directing users' attention 
towards important data and clarifying what the graphs 
are representing. 

Choice of sentences . A sentence with negative 
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wording may be much stronger than its alternative 
affirmative woring (Wright, 1977). For example, 
saying "Never use a subject sample size smaller than 
five" is more emphatic than saying "Choose a sample 
size of five or larger." Furthermore, Wason (1965) 
found that negative sentences are more effective at 
correcting a reader's expectations than positive 
sentences. However, because readers have an easier 
time with text that is written using simple, active, 
and affirmative statements, negative wording should be 
used sparingly when emphasis is required (e.g., 
findings or directions which conflict with user 
expectations) . 

Adjunct questions . Questions at the beginning of 
text help the reader to recall and comprehend specific 
information while questions after the text facilitate 
recall and comprehension of a broader nature (Rothkopf, 
1965). Washburne (1927) recommended using questions 
after a table or chart to emphasize its chief features. 

Color coding . Schutz (1961b) found that, in 
general, color coding improved performance over the 
black-and-white code, especially for multiple line 
graphs. Shontz and his associates reported that color 
coding works most effectively when (1) many categories 



49 



48 



of information are to be coded and (2) highly 
discriminate colors are used. Although color coding 
may add to the effectiveness of multiple line graphs, 
the fact that most school sites do not have color 
monitors may restrict its use. 

Bar charts. .To make bar charts easier to read, 
Culbertson and his associates (1959) recommend that the 
elements of bar charts should be labeled directly 
rather than indirectly by key or grid. In other words, 
labels for the bars should be located immediately next 
to the bars or within them. They also noted that 
horizontal bar charts may be preferable because they 
have more room for direct labels. 

Line graphs . The four most typical mistakes made 
when constructing graphs are the following: 
(1) something on the graph is not explained; (2) items 
on the graph cannot easily be distinguished due to the 
design or size of the graph; (3) mistakes exist such as 
incorrect spacing of tick marks, mislabeling, items 
omitted, or wrong scales; and (4) some aspect of the 
graph is missing or partially missing due to poor 
reproduction (Cleveland, 1985) . 

To avoid these errors, Cleveland (1985) recommends 
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the following; 

1. Make the data stand out. 

2. Do not clutter the data region. 

3. Do not overdo the. number of tick marks. 

4. Use a reference line when there is an important 
value that must be seen across the entire graph, but do 
not let the line interfere with the data. 

5. Do not let labels in the data region interfere 
with data display. 

6. Put keys just outside the data region and notes 
in the legend or text. 

7. Make superimposed data and plotting symbols 
visually distinguishable. 

8. Preserve visual clarity when reducing or 
reproducing. 

9. Choose the range of the tick marks to include 
or nearly include the range of data (e.g. if scores 
range from 200 to 800, so should the tick marks). 

10. Choosj the scales so that the data fill up as 
much of the data region as possible. 

11. Do not always include zero in a scale showing 
magnitude and use a logarithmic scale when it is 
important to understand percent or multiplicative 
factors . 
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12. Try to avoid scale breaks. 

13. If a scale break is absolutely necessary, do 
not connect numerical values on two sides of a break. 

Pictoaraohs . Research found the following 
characteristics to be essential to effective 
pictographs: (1) pictorial symbols should be self- 
explanatory and easily differentiated from one another, 
and (2) quantity is better represented by number of 
symbols than size of symbols (Neurath, 1944). 

Tables . Ehrenberg (1975) found that tables can be 
made easier to read by following these simple 
principles: 

1. Round numbers to two significant digits. (This 
facilitates mental arithmetic.) 

2. Provide row and column averages. 

3. Use columns for the most important comparisons. 
(Figures are easier to compare across columns.) 

4. Order rows and columns by size of numbers 
rather than by alphabetical order. 

5. Set columns and rows compactly. (Do not space 
them out to fit the ;age.) 

General Implications 

The following recommendations for the design and 
implementation of MLES are derived from research on how 
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data representation affects users' comprehension and 
use of data: 

1. Use presentation formats that require as little 
transformation as possible. In other words, present 
the information in the most straightforward and 
applicable form possible. 

2. Be consistent with representation styles. This 
means stick to one or a few types of graphs or tables 
throughout the system's displays. 

3. Represent the information in a format that 
allows the user to organize and conceptualize the 
information in terms of a small number of chunks or 
categories (no more than five) . 

4. Typical user errors in comprehension of 
statistical data might be avoided by including text 
with such findings that explicitly states their 
meanings and limitations. 

5. Use adjectival information rather than 
percentage information when accuracy of comprehension 
is essential. For example, for greater comprehension 
on the part of the user, use words such as "performs 
well in comparison to one's peers" rather than 
"performs in the 70th percentile." 

6. Use numerical rather than qualitative values 
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when comparisons between pieces of information are 
necessary. Although numerical values are not as easily 
understood as nonnumerical ones, they are easier for 
people to compare. For instance, a person can more 
easily compare the numbers "four" and "five" than the 
words !? fine" and "good." It should be noted that 
numbers may lead people to believe that there are more 
distinctive differences between individuals or groups 
than really exist. 

7. No one format has been found superior to all 
other formats, but it has been found that (1) a tabular 
representation is best where the task requires the 
simple retrieval of data, (2) bar charts are btJt for 
representing static comparisons, (3) line graphs are 
best for representing dynamic comparisons (trends) , and 
(4) pictographs are best for creating rapid and general 
impressions. 

8. Graphs and tables should use headings which are 
informative and unambiguous. 

9. Sentences should use negative rather than 
positive wording to provide special emphasis (e.g., 
when stating findings or directions that conflict with 
user expectations) . For example, if a user expects 
that low math achievement scores will be accompanied by 



54 



53 



low math motivation scores and this is not supported by 
the data, the system might state "Reject: math 
motivation and achievement are not related" rather than 
"math motivation and math achievement have an 
insignificant relationship." 

10. Questions that follow a graph or table may help 
users to better comprehend the data, derive 
implications for educational decision-making from the 
data, and decide whether further evaluation is 
necessary. 

11. If used, color coding should be used for making 
lines on multiple line graphs more discriminable. 
Further, the colors used must be easily differentiated 
from each other. 

12. Bar charts should be labeled directly rather 
than by key or grid. 

13. Line graphs should (1) be accompanied by 
supporting explanatory text, (2) use continuous scales 
which capture the range of actual rather than possible 
data, and (3) clearly display data and plotting 
symbols. 

14. Pictographs should be drawn with self- 
explanatory pictorial symbols that are easily 
differentiated. In addition, the amount rather than 
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the size of symbols should be used to represent 
quantity. 

15. Tables should (1) contain numbers rounded to 
two significant digits, (2) provide row and column 
averages, (3) place most important comparisons next to 
each other across columns, (4) order rows and columns 
by size of numbers, and (5) compactly set columns and 
rows. 

Computers and User Friendliness 

Introduction 

User friendliness is a term which refers to 
computer programs that are easy to use, tolerant of 
operator errors, and easy to learn. Research suggests 
that such programs are preferred by most users, are 
more time efficient, increase the likelihood of users 
accomplishing their goals, reduce the number of errors 
made by users, and reduce the amount of training time 
that is necessary for learning how to use the program 
(Simpson, 1985) . Recommendations for how to develop a 
user-friendly computer program come from the literature 
on software development concerned with computer-human 
interface. This section of the paper summarizes the 
major findings and recommendations that relate to 
developing a user-friendly program. 
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Recommendations fp r Developing a User-Friendiv Program 
The recommendations found in the user-friendliness 
literature reviewed tended to fall into these six 
categories: defining the user, minimizing the user's 
work, feedback, error detection and correction, 
legibility, user control, and principles of design. 

Defining the use r. Identifying and defining a 
program's users is a critical first requirement to 
assure user-friendliness. Audiences can consist of 
persons of varied skill levels (e.g., computer expert 
and computer naive) . Below are some recommendations 
based on knowing one's audience. 

1. Match the program to the user's skill level. 
If the users range in skill levels, two approaches 
developers might choose among are write for the lowest 
common denominator and provide different features for 
different users (Simpson, 1985). 

2. Use vocabulary that is at the user's level and 
common to his or her environment (Simpson, 1984) . 

3. Anticipate the environment in which the user 
will use the system. The main factors to consider are 
light, noise, and distractions. If the environment is 
unknown, it is best to assume a bright, loud, and 
distracting one, which would be best accommodated by a 
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program that has high contrast, is not dependent on 
meaningful sounds (alerting signals) , and has the 
capability to be stopped temporarily at any point and 
then later started up where the user left off (Simpson, 
1985) . 

Minimizing unneces sary labor . A us ^.r- friendly 
program attempts to minimize both the unnecessary 
mental and physical work required by the user in order 
to allow the user to concentrate his or her efforts on 
the purpose of the program. Mental work is the work 
involved in recalling commands or data, performing 
mental calculations, deciding what action to take next, 
and inferring meaning from the data reported. Physical 
work is the work involved in flipping switches, pushing 
keys, and changing disks. The following 
recommendations have been made in order to reduce 
memory and typing demands: 

1. Be consistent with input and output routines 
(data entry and screen displays) . Consistency makes a 
program easier to learn and results in fewer errors 
after one has learned it (Simpson, 1985; Gaines & 
Facey, 1975) . 

2. Provide prompting. Tell the user explicitly 
what to do next (Friend & Milokovic, 1984; Simpson, 
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1985) . 

3. Use a menu driven program (Simpson, 1984). 

4. Keep computer messages as concise as possible 
(Friend & Milojkovic, 1984; Simpson, 1984; Simpson, 
1985) . 

5* Design documentation manuals so users can 
locate information speedily. For instance, include 
content lists, indexes, page headings, and section 
headings. Also, communicate how to carry out 
procedures by a list of steps rather than by prose 
(e.g., sentences and paragraghs) (Wright, 1983). 

Feedback . Feedback is information that the 
computer provides to a user indicating that something 
he or she has done has had an effect on the computer. 
Feedback can confirm that the user has typed the 
appropriate characters or used the correct command. 
Feedback can also inform the user that he or she has 
made an error. Furthermore, feedback can let the user 
know when the computer is performing a function that 
must finish before the user can continue. The 
following are recommendations for how to produce 
feedback that will keep users on track: 

1. Display typed characters on screen (echoing) in 
order to inform users what input the computer is 
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receiving (Friend & Milojkovic, 1984). 

2. Validate data on entry (Gaines & Facey, 1975). 
Use response classifications such as correct, 
incorrect, or unrecognizable (Friend & Milojkovic, 
1984) . 

3. Give users feedback when their inputs will 

alter data or cause the*^6gr^m~"tsrlJranehH^ 

Milojkovic, 1984) . 

4. If the program requires a delay of five or more 
seconds, a message should appear on the screen telling 
the user what is going on (Simpson, 1985) . 

Error detection an d correction . Error tolerance 
is a central characteristic of user-friendly programs. 
Programs are error tolerant when they guard against a 
variety of input errors and allow users to easily 
correct mistakes that they have made. The following 
are recommendations for insuring error tolerance: 

1. Anticipate possible errors, check for them, and 
protect against them (Simpson, 1984). 

2. When an error is detected, alert the user and 
tell him or her how to recover. Alerting signals must 
differ from the customary background (e.g., flashing 
messages or beeps) (Simpson, 1984) . 

3. All for erasures and response editing (Friend 
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& Milojkovic, 1984; Gaines & Facey, 1975? Simpson 1984, 
1985) . 

4. Have users signal the end of their inputs by 
typing a "terminator" such, as the enter key or the 
return key; this will allow them to confirm or edit 
their responses (Friend & Milojkovic, 1984) . 

5. Enable users to make cor ections by re-entering 
data or backtracking with the cursor (Gaines & Facey, 
1975) . 

Screen display desi gn and legibility . Although 
screen layout is partly art and partly science, 
development can profit from observing the following 
recommendations : 

1. Do not crowd the screen displays (Simpson, 
1984) . 

2. Move from one screen display to another 
(paging) rather than scrolling (Simpson, 1984). 

3. For greatest readability text should be in 
upper and lower case letters with double spacing 
between lines (Tinker, 1965) . 

4. Most displays require a title centered at the 
top of the screen telling the user at what he or she is 
looking (Simpson, 1984) . 

5. Identify the different types of information 



that will need to be displayed on the screen (e.g., 
title block, numerical information, error-message line, 
and prompt line) and allocate a consistent screen area 
for each information category. Further, assure that 
the information on the screens do not stray from their 
assigned areas. Finally, if possible, separate each 
area oi the screen from the next by at least three rows 
or columns of blank spaces (Simpson, 1984). 

User control . Palme (1983, reports that computer 
users who are inexperienced and insecure prefer 
software which is fairly restrictive and guides them 
along predetermined paths with carefully worded 
questions. But as their experience grows, they become 
frustrated by the restrictiveness of the same software. 
He found that the more experienced the users became, 
the more they wanted the freedom to decide what to do 
and the ability to add new facilities to the software 
(e.g, macros, complex sequences of commands) . 
Therefore, the ideal software must accommodate users 
with varied amounts of experience and allow the users 
to decide how to use the computer. In order to 
increase the user's control over operating a program, 
the following recommendations have been made: 

1. Sustain operator orientation. In other words, 
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always inform the users where they are in the program 
and where they can go (Simpson, 1985) . 

2. Give the user the control for deciding when to 
move on in the program, what to save, and when to quit 
(Palme, 1983). 

3. Always make help available to users on line as 
well as with the aid of documentation. Provide both a 
general help facility (e.g., possible commands) and a 
specific help facility that, explains what the computer 
expects at that moment (Palme, 1983). 

4. Allow users to move from a menu-mode to a 
command-mode (Palme, 1983) . 

5. Allow users to save a series of commands 
(Palme, 1983). 

Principles of design . As mentioned earlier in 
this section, understanding the prospective users of a 
program is an important first step in developing a 
user- friendly program. Along with identifying 
prospective users and their needs, three other design 
principles have been recommended by Gould and Lewis 
(1983) : 

1. A panel of prospective users (e.g., teachers 
and principals) should work closely with the design 
tean during the early formulation stages. 
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2. Early in the development process, prospective 
users should actually use prototypes, and their 
performance and reactions should be measured. 

3. When problems are, found in user testing, they 
should be fixed in an iterative fashion. This means 
there should be a cycle of design, test and measure, 
and redesign, until each problem is solved. 

Conclusion 

This paper has reviewed the research literature 
relevant to matching an information system's design and 
capabilities with teachers and schools' organizational 
behaviors and needs, summarizing the findings of 
empirical studies and the experience-based 
recommendations of experts in a variety of disciplines. 
By going beyond the literature of educational 
information systems, we hoped to capitalize on (rather 
than reinvent) research conducted in other fields (e.g, 
management information systems, graphic design, and 
decision-making) . Our review of these various 
disciplines revealed considerable consistency in 
lessons learned and insights gleaned. From this we 
have derived 10 major principles for creating an 
effective, efficient, and attractive instructional 
information system. Each of these principles, 
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presented below, is followed by a check list of 
questions to aid in its application. 
Guiding Principles and Check Lists for a Successful 
Information System 

1. Create a system relevant to user needs and 
preferences. 

Have the system's primary users been identified? 

Who are they (e.g., primary or secondary teachers, 
principals, and school board members)? 

Have users clarified the role of the -system in 

their short and long term planning (e.g. , to increase 
efficiency in planning, monitoring and problem 
spotting, to improve student achievement, and to 
increase equality of opportunity to learn)? Is the 
system designed to directly support these roles? 

Have teachers and the school articulated short and 

long term educational priorities? Probe for goals in 
the areas of curriculum, extra curricular activities, 
student learning and development, teach :r control and 
authority, and motivational aspects of learning (e.g., 
instructional variety and self-esteem) . Is the system 
sensitive to these priorities? 

Does the system contain information that users 

feel will support teacher and school planning? Probe 
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in the following areas: 

1. district and school priorities and 
requirements, 

2. pupil characteristics (e.g., demographic 
information, academic needs, attendance, 
stability) , 

3. pupil outcomes (performance on standardized 
tests, curriculum based tests, attitude 
measures, career plans) , 

4. parent/ community characteristics, and 

5. available resources? 

Does the design of the system account for the 

following attributes of prospective users: 

1. computer skill level, 

2. skill and experience in analyzing and 
interpreting data, 

3. vocabulary, and 

4. graphic/reporting preferences? 

When will the system's data be needed (e.g. , 

before school year begins)? Will the data be entered 
and be available before users need to use it? 

Is the system design responsive to the ways users' 

conceptualize problems and their solution? 

Have the ways the system will support and 

facilitate teachers and schools routine functioning 
been clearly identified? 

Have users been centrally involved at each stage 
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of planning and implementation and been given adequate 
opportunity to review and suggest improvements (e.g., 
through steering committees, repeated prototype 
trials)? 

2. Account for the organisational and political 
climate* 

Have special sources of resist re to implementing 

such a system been identified (e.g., staff resistance 
to change, competition for financial resources, 
computer or data phobia, fear of misuse/ abuse) and 
addressed? 

Have influential actors in decision-making, 

resource allocation and leadership roles been 
identified and convinced of the importance of the 
project? 

Does the system avoid using the data for teacher 

or staff evaluation/accountability, and instead focus 
on facilitating improvement? 

Is student and teacher privacy protected with the 

use of passwords? 

3. Minimise the memory load of commands and procedures 
needed to learn and operate the system (make learning 
easy) * 

Are input and output routines consistent? 
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Are prompts always presented with directions about 

what users should do next? 

Are menus of possible inquiries offered to users? 

Is on line documentation or help available to 

users? 

Does the system detect errors and allow for easy 

correction? 

Are computer messages concise and clear? 

Does the system use simple and nontechnical 

language? 

4. Minimize the amount of information that the users 
need to process at any given moment. Make processing 
easy and avoid information overload. 

Can information reported on a single screen or 

page be organized into a maximum of five to seven 
chunks? 

Do display screens have the following 

characteristics : 

X. uncrowded text and graphs, 

2. paging rather than scrolling, 

3. upper and lower case letters with double 
spacing between lines, and 

4. a centered title at the top of the screen that 
tells the user what is being viewed? 

5. Do reports and displays facilitate users' 
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comprehension and correct interpretation of the data 
provided? 

Are the tables and graphs selected to ease 

comprehension and utility: 

1. tables for simple retrieval, 

2. bar charts for static comparisons, and 

3. single or multiple line graphs for dynamic 
comparisons (trends)? 

Are graphs and tables marked with clear headings 

in upper and lower case text? 

Are statistical data accompanied with text that 

explicitly states their meanings and limitations? 

Are graphs and tables followed with questions to 

encourage active processing and to direct users 
attention toward specific applications or further data 
collection? 

6. Insure the accuracy of data. 

Does the system have built in checks for out of 

range data? 

Does the system offer same day scoring with scan 

machines that enter the data directly into the system? 

Does the system have safeguards to prevent 

inappropriate analyses and interpretation (e.g., 
analyses that are based on too small an n, calculations 
that are technically incorrect)? 
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7. Does the system encourage users to confront their 
expectations? 

Are users asked to state their expectations 

regarding possible relationships, trends, or relative 
standings? 

Are users informed if their expectations are 

supported or not supported by the data? 

8. Maintain users sense of control when using the 
system. 

Are users given the opportunity to edit and 

confirm inputs by typing a terminator such as the enter 
key? 

Are users provided with feedback after every data 

entry and computer command, and during computer delays? 

Are users given control to decide when to move on 

in the program, what to save, and when to quit? 

Does the system allow users to generate their own 

unique queries? 

9. Insure that user attitudes are positive toward using 
the system and incorporating its results into their 
planning. 

Have users been actively involved in the system's 

development? 

Do users feel a sense of ownership of the system? 
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Do users perceive a relationship between the 

system and their school responsibilities? 

Do users have confidence in the accuracy and 

objectivity of the data? 

Is there sufficient training to assure that users 

are comfortable with the process and able to operate 
the system? 

Do users believe their information system 

facilitates better planning? To increase such beliefs, 

Has training presented concrete examples of 

how the system has been used to improve planning? 

Are users encouraged to consider findings 

which challenge the status quo and to imagine 
their implications for the future, even if they 
are not immediately feasible? 

Are there opportunities to modify the system 

to help it better meet teacher needs? 
10. Provide continued support end advocacy for the 
system's use with users after they have completed the 
training. 

Do users know where they can receive further 

assistance for using the system and analyzing its. data 
after the training has finished? 

Are users consulted and informed before changes 
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are made in the system? 

_ Is there an influential "idea champion" in every 

local setting who actively advocates the system's 
capabilities and the importance of its results to 
users? 

Are users given the opportunity to evaluate the 

system (e.g., electronic suggestion box or written 
evaluations) ? 
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